Skip to content

Fix: Make coverage error messages framework-agnostic#1990

Merged
Saga4 merged 1 commit intomainfrom
fix/coverage-utils-framework-agnostic-messages
Apr 4, 2026
Merged

Fix: Make coverage error messages framework-agnostic#1990
Saga4 merged 1 commit intomainfrom
fix/coverage-utils-framework-agnostic-messages

Conversation

@mohammedahmed18
Copy link
Copy Markdown
Contributor

@mohammedahmed18 mohammedahmed18 commented Apr 4, 2026

Problem

Error messages in coverage_utils.py hardcoded "Jest" even when the test framework was Vitest, causing confusion in logs (e.g., "Jest coverage file not found" when using Vitest).

Root Cause

The JestCoverageUtils class is used for both Jest AND Vitest since they share the same Istanbul/v8 coverage format. Error messages should be framework-agnostic, not hardcode "Jest".

Solution

Changed 4 error message strings to be framework-agnostic:

  • ❌ "Jest coverage file not found"

  • ✅ "JavaScript coverage file not found"

  • ❌ "Failed to parse Jest coverage file"

  • ✅ "Failed to parse JavaScript coverage file"

  • ❌ "No coverage data found for X in Jest coverage"

  • ✅ "No coverage data found for X in JavaScript coverage"

  • ❌ "Function X not found in Jest fnMap"

  • ✅ "Function X not found in JavaScript fnMap"

Testing

  • ✅ Added 2 new unit tests in tests/verification/test_coverage_utils_framework_agnostic.py
  • ✅ All tests pass
  • ✅ uv run prek passes with no linting/type errors

Affected Logs

This fix improves clarity for 7 logs from the Apr 4 00:50 optimization batch:

  • 37e5a406-15b1-40c9-80aa-76b5c239ee41
  • 735555fa-67c2-4691-a308-66cd71c94535
  • 940dfe80-dae9-43df-9576-cf23a0e57352
  • c1e1de0e-9f7e-4281-990a-57d7ad47d20a
  • dbec6c33-ef80-479f-ae07-2f3b0acb1441
  • de96b1ab-8db5-4e88-a4b8-458dcd79f69c
  • fcf08c6b-9c81-4754-a860-e4119686e654

Note

The actual test failures in these logs (tests not passing for original code) are non-deterministic LLM test generation issues - expected behavior per workflow, not systematic bugs requiring fixes.

@mohammedahmed18 @claude
Fix: Make coverage error messages framework-agnostic
933a260 
Error messages in coverage_utils.py hardcoded "Jest" even when the
test framework was Vitest. This caused confusion in logs when Vitest
tests failed (e.g., "Jest coverage file not found" when using Vitest).

The JestCoverageUtils class is used for both Jest and Vitest since
they share the same Istanbul/v8 coverage format. Error messages
should be framework-agnostic.

Changes:
- "Jest coverage file not found" → "JavaScript coverage file not found"
- "Failed to parse Jest coverage file" → "Failed to parse JavaScript coverage file"
- "No coverage data found for X in Jest coverage" → "No coverage data found for X in JavaScript coverage"
- "Function X not found in Jest fnMap" → "Function X not found in JavaScript fnMap"

Affected trace IDs: 37e5a406, 735555fa, 940dfe80, c1e1de0e, dbec6c33, de96b1ab, fcf08c6b (7 logs from Apr 4 00:50 batch)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
@claude
Copy link
Copy Markdown
Contributor

claude bot commented Apr 4, 2026 •
edited
Loading

Claude finished @mohammedahmed18's task in 2m 1s —— View job

PR Review Summary

Prek Checks

uv run prek run --from-ref origin/main passes with no issues.

Code Review

Production change (codeflash/verification/coverage_utils.py): Correct and minimal — 4 log message strings updated from "Jest ..." to "JavaScript ...". No logic changes. LGTM.

Test file (tests/verification/test_coverage_utils_framework_agnostic.py): A few issues:

  1. "EMPTY" is not a valid CoverageStatus value (tests/verification/test_coverage_utils_framework_agnostic.py:43,82). CoverageStatus only has NOT_FOUND and PARSED_SUCCESSFULLY — there is no EMPTY member. The assertions result.status.name in ("NOT_FOUND", "EMPTY") pass in practice only because create_empty() returns CoverageStatus.NOT_FOUND, but "EMPTY" is dead code here. Should be == CoverageStatus.NOT_FOUND (matching the pattern in tests/languages/javascript/test_vitest_coverage_exclusions.py:68). Fix this →

  2. Unused import (tests/verification/test_coverage_utils_framework_agnostic.py:3): import tempfile is imported but never used — tmp_path is used correctly instead. Fix this →

  3. Weak test assertions (tests/verification/test_coverage_utils_framework_agnostic.py:50-58): The first test wraps the key assertion in if coverage_messages:, so if no log messages are captured at all (e.g., logger propagation issue), the test passes vacuously even if the fix were reverted. Should unconditionally assert that a message was logged, then check its content.

  4. Missing return type annotations on both test methods — mypy flags these (no-untyped-def). Should be def test_...(self, ...) -> None:.

Duplicate Detection

No duplicates detected. The PR modifies string literals only; no new logic is introduced.

Last updated: 2026-04-04

@Saga4 Saga4 merged commit 755d0f2 into main Apr 4, 2026
29 of 31 checks passed
@Saga4 Saga4 deleted the fix/coverage-utils-framework-agnostic-messages branch April 4, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Saga4 Saga4 Saga4 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mohammedahmed18 @Saga4